In [ ]:
!pip install auto-sklearn -q
!pip install openml -q
!pip install PipelineProfiler -q
!pip install H2O -q
!pip install gama -q
!pip install transformers -q
!pip install datasets -q
# restart runtime after installing
     |████████████████████████████████| 6.3 MB 4.7 MB/s 
     |████████████████████████████████| 28.5 MB 54 kB/s 
     |████████████████████████████████| 22.3 MB 35.8 MB/s 
     |████████████████████████████████| 722 kB 40.9 MB/s 
     |████████████████████████████████| 4.2 MB 48.8 MB/s 
     |████████████████████████████████| 4.0 MB 31.4 MB/s 
     |████████████████████████████████| 208 kB 50.0 MB/s 
     |████████████████████████████████| 973 kB 35.4 MB/s 
     |████████████████████████████████| 125 kB 46.9 MB/s 
     |████████████████████████████████| 45 kB 3.5 MB/s 
  Building wheel for auto-sklearn (setup.py) ... done
  Building wheel for pynisher (setup.py) ... done
  Building wheel for liac-arff (setup.py) ... done
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
gym 0.17.3 requires cloudpickle<1.7.0,>=1.2.0, but you have cloudpickle 2.0.0 which is incompatible.
albumentations 0.1.12 requires imgaug<0.2.7,>=0.2.5, but you have imgaug 0.2.9 which is incompatible.
     |████████████████████████████████| 119 kB 5.2 MB/s 
     |████████████████████████████████| 75 kB 4.1 MB/s 
  Building wheel for openml (setup.py) ... done
     |████████████████████████████████| 880 kB 5.1 MB/s 
     |████████████████████████████████| 175.8 MB 22 kB/s 
  Building wheel for H2O (setup.py) ... done
     |████████████████████████████████| 102 kB 6.5 MB/s 
     |████████████████████████████████| 10.1 MB 17.1 MB/s 
     |████████████████████████████████| 82 kB 375 kB/s 
     |████████████████████████████████| 97 kB 6.5 MB/s 
     |████████████████████████████████| 743 kB 40.9 MB/s 
  Building wheel for stopit (setup.py) ... done
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
google-colab 1.0.0 requires pandas~=1.1.0; python_version >= "3.0", but you have pandas 1.0.5 which is incompatible.
     |████████████████████████████████| 3.1 MB 5.3 MB/s 
     |████████████████████████████████| 895 kB 38.2 MB/s 
     |████████████████████████████████| 59 kB 6.3 MB/s 
     |████████████████████████████████| 596 kB 36.7 MB/s 
     |████████████████████████████████| 3.3 MB 39.4 MB/s 
     |████████████████████████████████| 290 kB 4.9 MB/s 
     |████████████████████████████████| 243 kB 40.3 MB/s 
     |████████████████████████████████| 1.1 MB 38.9 MB/s 
     |████████████████████████████████| 271 kB 48.1 MB/s 
     |████████████████████████████████| 160 kB 45.8 MB/s 
     |████████████████████████████████| 192 kB 50.3 MB/s 

Week 6 - Transfer Learning and AutoML

Standing on the shoulders of giants. This week we will be using very large models trained om huge datasets. Using Transfer Learning you will be able to fit these pre-trained models on your data. This will save you a lot of time and processing power. Furthermore we'll walk you through some extremely useful AutoML libraries. Knowing how to use that will potentially take a lot of work out of your hands. Let's start with some AutoML!

In [ ]:
# First we are going to take a look at the autosklearn library
# Restart runtime if it craches 
import autosklearn.classification
import sklearn.model_selection
import sklearn.datasets
import sklearn.metrics
import PipelineProfiler

# load the data set and split
X, y = sklearn.datasets.load_digits(return_X_y=True)
X_train, X_test, y_train, y_test = \
    sklearn.model_selection.train_test_split(X, y, random_state=1)

Exercise 1

A) Have a look at the shape of the data

B) What are the numbers suposed to mean in each instance?

C) Look up https://scikit-learn.org/stable/modules/generated/sklearn.datasets.load_digits.html

D) Create a plot that visualises an instance of X

In [ ]:
print(X[0])
[ 0.  0.  5. 13.  9.  1.  0.  0.  0.  0. 13. 15. 10. 15.  5.  0.  0.  3.
 15.  2.  0. 11.  8.  0.  0.  4. 12.  0.  0.  8.  8.  0.  0.  5.  8.  0.
  0.  9.  8.  0.  0.  4. 11.  0.  1. 12.  7.  0.  0.  2. 14.  5. 10. 12.
  0.  0.  0.  0.  6. 13. 10.  0.  0.  0.]
In [ ]:
digits = sklearn.datasets.load_digits()
print(digits.images[0])
import matplotlib.pyplot as plt
plt.gray()
plt.matshow(digits.images[0])
plt.show()
print('Label:',y[0])
[[ 0.  0.  5. 13.  9.  1.  0.  0.]
 [ 0.  0. 13. 15. 10. 15.  5.  0.]
 [ 0.  3. 15.  2.  0. 11.  8.  0.]
 [ 0.  4. 12.  0.  0.  8.  8.  0.]
 [ 0.  5.  8.  0.  0.  9.  8.  0.]
 [ 0.  4. 11.  0.  1. 12.  7.  0.]
 [ 0.  2. 14.  5. 10. 12.  0.  0.]
 [ 0.  0.  6. 13. 10.  0.  0.  0.]]
<Figure size 432x288 with 0 Axes>
Label: 0

Alright, now we are going to train a classifier using auto-sklearn to classify each number. Run the following code. It should take around a minute and you should get an accuracy of >.98

In [ ]:
automl = autosklearn.classification.AutoSklearnClassifier(
    time_left_for_this_task=60, # sec., how long should this seed fit process run
    per_run_time_limit=15, # sec., each model may only take this long before it's killed
)

automl.fit(X_train, y_train)
y_hat = automl.predict(X_test)
print("Accuracy score", sklearn.metrics.accuracy_score(y_test, y_hat))
Accuracy score 0.9888888888888889

Okay.... Looks cool! But how do we know now what our model looks like? And on what parameters it is tuned?

Exercise 2

A) Look up in the manual how to inspect your results https://automl.github.io/auto-sklearn/master/manual.html

B) What model perfroms best?

C) Find the parameters set for you model

D) Look up PipelineProfiler and run it to inspect your trained autoML

In [ ]:
# Print the final ensemble constructed by auto-sklearn.
print(automl.show_models())
predictions = automl.predict(X_test)
# Print statistics about the auto-sklearn run such as number of
# iterations, number of models failed with a time out.
print(automl.sprint_statistics())
print("Accuracy score", sklearn.metrics.accuracy_score(y_test, predictions))
[(0.220000, SimpleClassificationPipeline({'balancing:strategy': 'none', 'classifier:__choice__': 'libsvm_svc', 'data_preprocessor:__choice__': 'feature_type', 'feature_preprocessor:__choice__': 'feature_agglomeration', 'classifier:libsvm_svc:C': 8776.614453785322, 'classifier:libsvm_svc:gamma': 2.6166845238639262, 'classifier:libsvm_svc:kernel': 'poly', 'classifier:libsvm_svc:max_iter': -1, 'classifier:libsvm_svc:shrinking': 'False', 'classifier:libsvm_svc:tol': 4.6482002538704e-05, 'data_preprocessor:feature_type:categorical_transformer:categorical_encoding:__choice__': 'one_hot_encoding', 'data_preprocessor:feature_type:categorical_transformer:category_coalescence:__choice__': 'no_coalescense', 'data_preprocessor:feature_type:numerical_transformer:imputation:strategy': 'median', 'data_preprocessor:feature_type:numerical_transformer:rescaling:__choice__': 'none', 'feature_preprocessor:feature_agglomeration:affinity': 'manhattan', 'feature_preprocessor:feature_agglomeration:linkage': 'average', 'feature_preprocessor:feature_agglomeration:n_clusters': 329, 'feature_preprocessor:feature_agglomeration:pooling_func': 'max', 'classifier:libsvm_svc:coef0': -0.33548507886436374, 'classifier:libsvm_svc:degree': 2},
dataset_properties={
  'task': 2,
  'sparse': False,
  'multilabel': False,
  'multiclass': True,
  'target_type': 'classification',
  'signed': False})),
(0.160000, SimpleClassificationPipeline({'balancing:strategy': 'weighting', 'classifier:__choice__': 'libsvm_svc', 'data_preprocessor:__choice__': 'feature_type', 'feature_preprocessor:__choice__': 'no_preprocessing', 'classifier:libsvm_svc:C': 21.59109048521139, 'classifier:libsvm_svc:gamma': 5.060493057005212, 'classifier:libsvm_svc:kernel': 'rbf', 'classifier:libsvm_svc:max_iter': -1, 'classifier:libsvm_svc:shrinking': 'False', 'classifier:libsvm_svc:tol': 0.00012027336497045934, 'data_preprocessor:feature_type:categorical_transformer:categorical_encoding:__choice__': 'one_hot_encoding', 'data_preprocessor:feature_type:categorical_transformer:category_coalescence:__choice__': 'minority_coalescer', 'data_preprocessor:feature_type:numerical_transformer:imputation:strategy': 'most_frequent', 'data_preprocessor:feature_type:numerical_transformer:rescaling:__choice__': 'normalize', 'data_preprocessor:feature_type:categorical_transformer:category_coalescence:minority_coalescer:minimum_fraction': 0.017236598742666608},
dataset_properties={
  'task': 2,
  'sparse': False,
  'multilabel': False,
  'multiclass': True,
  'target_type': 'classification',
  'signed': False})),
(0.140000, SimpleClassificationPipeline({'balancing:strategy': 'none', 'classifier:__choice__': 'gradient_boosting', 'data_preprocessor:__choice__': 'feature_type', 'feature_preprocessor:__choice__': 'select_percentile_classification', 'classifier:gradient_boosting:early_stop': 'valid', 'classifier:gradient_boosting:l2_regularization': 1.7049772538367706e-08, 'classifier:gradient_boosting:learning_rate': 0.0825755415435688, 'classifier:gradient_boosting:loss': 'auto', 'classifier:gradient_boosting:max_bins': 255, 'classifier:gradient_boosting:max_depth': 'None', 'classifier:gradient_boosting:max_leaf_nodes': 51, 'classifier:gradient_boosting:min_samples_leaf': 121, 'classifier:gradient_boosting:scoring': 'loss', 'classifier:gradient_boosting:tol': 1e-07, 'data_preprocessor:feature_type:categorical_transformer:categorical_encoding:__choice__': 'one_hot_encoding', 'data_preprocessor:feature_type:categorical_transformer:category_coalescence:__choice__': 'minority_coalescer', 'data_preprocessor:feature_type:numerical_transformer:imputation:strategy': 'most_frequent', 'data_preprocessor:feature_type:numerical_transformer:rescaling:__choice__': 'robust_scaler', 'feature_preprocessor:select_percentile_classification:percentile': 27.329670949267125, 'feature_preprocessor:select_percentile_classification:score_func': 'mutual_info', 'classifier:gradient_boosting:n_iter_no_change': 10, 'classifier:gradient_boosting:validation_fraction': 0.15078023719798528, 'data_preprocessor:feature_type:categorical_transformer:category_coalescence:minority_coalescer:minimum_fraction': 0.00011592043029831034, 'data_preprocessor:feature_type:numerical_transformer:rescaling:robust_scaler:q_max': 0.8101878073561093, 'data_preprocessor:feature_type:numerical_transformer:rescaling:robust_scaler:q_min': 0.23656284692369586},
dataset_properties={
  'task': 2,
  'sparse': False,
  'multilabel': False,
  'multiclass': True,
  'target_type': 'classification',
  'signed': False})),
(0.120000, SimpleClassificationPipeline({'balancing:strategy': 'none', 'classifier:__choice__': 'random_forest', 'data_preprocessor:__choice__': 'feature_type', 'feature_preprocessor:__choice__': 'feature_agglomeration', 'classifier:random_forest:bootstrap': 'False', 'classifier:random_forest:criterion': 'gini', 'classifier:random_forest:max_depth': 'None', 'classifier:random_forest:max_features': 0.9228410070967645, 'classifier:random_forest:max_leaf_nodes': 'None', 'classifier:random_forest:min_impurity_decrease': 0.0, 'classifier:random_forest:min_samples_leaf': 18, 'classifier:random_forest:min_samples_split': 3, 'classifier:random_forest:min_weight_fraction_leaf': 0.0, 'data_preprocessor:feature_type:categorical_transformer:categorical_encoding:__choice__': 'one_hot_encoding', 'data_preprocessor:feature_type:categorical_transformer:category_coalescence:__choice__': 'no_coalescense', 'data_preprocessor:feature_type:numerical_transformer:imputation:strategy': 'mean', 'data_preprocessor:feature_type:numerical_transformer:rescaling:__choice__': 'none', 'feature_preprocessor:feature_agglomeration:affinity': 'euclidean', 'feature_preprocessor:feature_agglomeration:linkage': 'ward', 'feature_preprocessor:feature_agglomeration:n_clusters': 202, 'feature_preprocessor:feature_agglomeration:pooling_func': 'mean'},
dataset_properties={
  'task': 2,
  'sparse': False,
  'multilabel': False,
  'multiclass': True,
  'target_type': 'classification',
  'signed': False})),
(0.100000, SimpleClassificationPipeline({'balancing:strategy': 'none', 'classifier:__choice__': 'passive_aggressive', 'data_preprocessor:__choice__': 'feature_type', 'feature_preprocessor:__choice__': 'polynomial', 'classifier:passive_aggressive:C': 0.0007163174331946707, 'classifier:passive_aggressive:average': 'False', 'classifier:passive_aggressive:fit_intercept': 'True', 'classifier:passive_aggressive:loss': 'hinge', 'classifier:passive_aggressive:tol': 1.0000041320668022e-05, 'data_preprocessor:feature_type:categorical_transformer:categorical_encoding:__choice__': 'one_hot_encoding', 'data_preprocessor:feature_type:categorical_transformer:category_coalescence:__choice__': 'minority_coalescer', 'data_preprocessor:feature_type:numerical_transformer:imputation:strategy': 'median', 'data_preprocessor:feature_type:numerical_transformer:rescaling:__choice__': 'minmax', 'feature_preprocessor:polynomial:degree': 2, 'feature_preprocessor:polynomial:include_bias': 'True', 'feature_preprocessor:polynomial:interaction_only': 'True', 'data_preprocessor:feature_type:categorical_transformer:category_coalescence:minority_coalescer:minimum_fraction': 0.0099235580062201},
dataset_properties={
  'task': 2,
  'sparse': False,
  'multilabel': False,
  'multiclass': True,
  'target_type': 'classification',
  'signed': False})),
(0.100000, SimpleClassificationPipeline({'balancing:strategy': 'weighting', 'classifier:__choice__': 'libsvm_svc', 'data_preprocessor:__choice__': 'feature_type', 'feature_preprocessor:__choice__': 'no_preprocessing', 'classifier:libsvm_svc:C': 575.2840839820392, 'classifier:libsvm_svc:gamma': 0.28251015893293174, 'classifier:libsvm_svc:kernel': 'rbf', 'classifier:libsvm_svc:max_iter': -1, 'classifier:libsvm_svc:shrinking': 'False', 'classifier:libsvm_svc:tol': 6.314728575433805e-05, 'data_preprocessor:feature_type:categorical_transformer:categorical_encoding:__choice__': 'one_hot_encoding', 'data_preprocessor:feature_type:categorical_transformer:category_coalescence:__choice__': 'no_coalescense', 'data_preprocessor:feature_type:numerical_transformer:imputation:strategy': 'median', 'data_preprocessor:feature_type:numerical_transformer:rescaling:__choice__': 'none'},
dataset_properties={
  'task': 2,
  'sparse': False,
  'multilabel': False,
  'multiclass': True,
  'target_type': 'classification',
  'signed': False})),
(0.080000, SimpleClassificationPipeline({'balancing:strategy': 'none', 'classifier:__choice__': 'random_forest', 'data_preprocessor:__choice__': 'feature_type', 'feature_preprocessor:__choice__': 'no_preprocessing', 'classifier:random_forest:bootstrap': 'True', 'classifier:random_forest:criterion': 'gini', 'classifier:random_forest:max_depth': 'None', 'classifier:random_forest:max_features': 0.5, 'classifier:random_forest:max_leaf_nodes': 'None', 'classifier:random_forest:min_impurity_decrease': 0.0, 'classifier:random_forest:min_samples_leaf': 1, 'classifier:random_forest:min_samples_split': 2, 'classifier:random_forest:min_weight_fraction_leaf': 0.0, 'data_preprocessor:feature_type:categorical_transformer:categorical_encoding:__choice__': 'one_hot_encoding', 'data_preprocessor:feature_type:categorical_transformer:category_coalescence:__choice__': 'minority_coalescer', 'data_preprocessor:feature_type:numerical_transformer:imputation:strategy': 'mean', 'data_preprocessor:feature_type:numerical_transformer:rescaling:__choice__': 'standardize', 'data_preprocessor:feature_type:categorical_transformer:category_coalescence:minority_coalescer:minimum_fraction': 0.01},
dataset_properties={
  'task': 2,
  'sparse': False,
  'multilabel': False,
  'multiclass': True,
  'target_type': 'classification',
  'signed': False})),
(0.080000, SimpleClassificationPipeline({'balancing:strategy': 'none', 'classifier:__choice__': 'mlp', 'data_preprocessor:__choice__': 'feature_type', 'feature_preprocessor:__choice__': 'liblinear_svc_preprocessor', 'classifier:mlp:activation': 'tanh', 'classifier:mlp:alpha': 3.198483470889531e-06, 'classifier:mlp:batch_size': 'auto', 'classifier:mlp:beta_1': 0.9, 'classifier:mlp:beta_2': 0.999, 'classifier:mlp:early_stopping': 'train', 'classifier:mlp:epsilon': 1e-08, 'classifier:mlp:hidden_layer_depth': 2, 'classifier:mlp:learning_rate_init': 0.006604847357173181, 'classifier:mlp:n_iter_no_change': 32, 'classifier:mlp:num_nodes_per_layer': 24, 'classifier:mlp:shuffle': 'True', 'classifier:mlp:solver': 'adam', 'classifier:mlp:tol': 0.0001, 'data_preprocessor:feature_type:categorical_transformer:categorical_encoding:__choice__': 'no_encoding', 'data_preprocessor:feature_type:categorical_transformer:category_coalescence:__choice__': 'no_coalescense', 'data_preprocessor:feature_type:numerical_transformer:imputation:strategy': 'median', 'data_preprocessor:feature_type:numerical_transformer:rescaling:__choice__': 'standardize', 'feature_preprocessor:liblinear_svc_preprocessor:C': 0.6755226057791226, 'feature_preprocessor:liblinear_svc_preprocessor:dual': 'False', 'feature_preprocessor:liblinear_svc_preprocessor:fit_intercept': 'True', 'feature_preprocessor:liblinear_svc_preprocessor:intercept_scaling': 1, 'feature_preprocessor:liblinear_svc_preprocessor:loss': 'squared_hinge', 'feature_preprocessor:liblinear_svc_preprocessor:multi_class': 'ovr', 'feature_preprocessor:liblinear_svc_preprocessor:penalty': 'l1', 'feature_preprocessor:liblinear_svc_preprocessor:tol': 0.00014091021190100242},
dataset_properties={
  'task': 2,
  'sparse': False,
  'multilabel': False,
  'multiclass': True,
  'target_type': 'classification',
  'signed': False})),
]
auto-sklearn results:
  Dataset name: 309a06ca-3705-11ec-80f8-0242ac1c0002
  Metric: accuracy
  Best validation score: 0.988764
  Number of target algorithm runs: 11
  Number of successful target algorithm runs: 8
  Number of crashed target algorithm runs: 0
  Number of target algorithms that exceeded the time limit: 2
  Number of target algorithms that exceeded the memory limit: 1

Accuracy score 0.9888888888888889
In [ ]:
profiler_data= PipelineProfiler.import_autosklearn(automl)
PipelineProfiler.plot_pipeline_matrix(profiler_data)

Next up we'll use the H2O AutoML library https://docs.h2o.ai/h2o/latest-stable/h2o-docs/automl.html

Below you find a classification example of using H2O AutoML. You can run the code to generate the results but this can take up to 30 minutes.

In [ ]:
import h2o
from h2o.automl import H2OAutoML

# Start the H2O cluster (locally)
h2o.init()

# Import a sample binary outcome train/test set into H2O
train = h2o.import_file("https://s3.amazonaws.com/erin-data/higgs/higgs_train_10k.csv")
test = h2o.import_file("https://s3.amazonaws.com/erin-data/higgs/higgs_test_5k.csv")

# Identify predictors and response
x = train.columns
y = "response"
x.remove(y)

# For binary classification, response should be a factor
train[y] = train[y].asfactor()
test[y] = test[y].asfactor()

# Run AutoML for 20 base models
aml = H2OAutoML(max_models=10, seed=1)
aml.train(x=x, y=y, training_frame=train)

# View the AutoML Leaderboard
lb = aml.leaderboard
lb.head(rows=lb.nrows)  # Print all rows instead of default (10 rows)

# The leader model is stored here
aml.leader
Checking whether there is an H2O instance running at http://localhost:54321 ..... not found.
Attempting to start a local H2O server...
  Java Version: openjdk version "11.0.11" 2021-04-20; OpenJDK Runtime Environment (build 11.0.11+9-Ubuntu-0ubuntu2.18.04); OpenJDK 64-Bit Server VM (build 11.0.11+9-Ubuntu-0ubuntu2.18.04, mixed mode, sharing)
  Starting server from /usr/local/lib/python3.7/dist-packages/h2o/backend/bin/h2o.jar
  Ice root: /tmp/tmpo2wsd_02
  JVM stdout: /tmp/tmpo2wsd_02/h2o_unknownUser_started_from_python.out
  JVM stderr: /tmp/tmpo2wsd_02/h2o_unknownUser_started_from_python.err
  Server is running at http://127.0.0.1:54321
Connecting to H2O server at http://127.0.0.1:54321 ... successful.
H2O_cluster_uptime: 02 secs
H2O_cluster_timezone: Etc/UTC
H2O_data_parsing_timezone: UTC
H2O_cluster_version: 3.34.0.3
H2O_cluster_version_age: 19 days
H2O_cluster_name: H2O_from_python_unknownUser_3eco9s
H2O_cluster_total_nodes: 1
H2O_cluster_free_memory: 3.172 Gb
H2O_cluster_total_cores: 2
H2O_cluster_allowed_cores: 2
H2O_cluster_status: locked, healthy
H2O_connection_url: http://127.0.0.1:54321
H2O_connection_proxy: {"http": null, "https": null}
H2O_internal_security: False
H2O_API_Extensions: Amazon S3, XGBoost, Algos, AutoML, Core V3, TargetEncoder, Core V4
Python_version: 3.7.12 final
Parse progress: |████████████████████████████████████████████████████████████████| (done) 100%
Parse progress: |████████████████████████████████████████████████████████████████| (done) 100%
AutoML progress: |███████████████████████████████████████████████████████████████| (done) 100%
Model Details
=============
H2OStackedEnsembleEstimator :  Stacked Ensemble
Model Key:  StackedEnsemble_AllModels_5_AutoML_1_20211027_90755

No model summary for this model

ModelMetricsBinomialGLM: stackedensemble
** Reported on train data. **

MSE: 0.08469572014178943
RMSE: 0.2910252912407948
LogLoss: 0.3106163519789774
Null degrees of freedom: 9999
Residual degrees of freedom: 9989
Null deviance: 13828.113387424315
Residual deviance: 6212.327039579549
AIC: 6234.327039579549
AUC: 0.9807508135820792
AUCPR: 0.9825777942887437
Gini: 0.9615016271641583

Confusion Matrix (Act/Pred) for max f1 @ threshold = 0.5044680777947245: 
0 1 Error Rate
0 0 4273.0 432.0 0.0918 (432.0/4705.0)
1 1 314.0 4981.0 0.0593 (314.0/5295.0)
2 Total 4587.0 5413.0 0.0746 (746.0/10000.0)
Maximum Metrics: Maximum metrics at their respective thresholds
metric threshold value idx
0 max f1 0.504468 0.930332 205.0
1 max f2 0.397284 0.956192 241.0
2 max f0point5 0.582793 0.941905 176.0
3 max accuracy 0.504468 0.925400 205.0
4 max precision 0.995037 1.000000 0.0
5 max recall 0.214485 1.000000 316.0
6 max specificity 0.995037 1.000000 0.0
7 max absolute_mcc 0.504468 0.850308 205.0
8 max min_per_class_accuracy 0.526593 0.922635 197.0
9 max mean_per_class_accuracy 0.507497 0.924465 204.0
10 max tns 0.995037 4705.000000 0.0
11 max fns 0.995037 5289.000000 0.0
12 max fps 0.004656 4705.000000 399.0
13 max tps 0.214485 5295.000000 316.0
14 max tnr 0.995037 1.000000 0.0
15 max fnr 0.995037 0.998867 0.0
16 max fpr 0.004656 1.000000 399.0
17 max tpr 0.214485 1.000000 316.0
Gains/Lift Table: Avg response rate: 52.95 %, avg score: 52.97 %
group cumulative_data_fraction lower_threshold lift cumulative_lift response_rate score cumulative_response_rate cumulative_score capture_rate cumulative_capture_rate gain cumulative_gain kolmogorov_smirnov
0 1 0.01 0.961094 1.888574 1.888574 1.000 0.969797 1.000000 0.969797 0.018886 0.018886 88.857413 88.857413 0.018886
1 2 0.02 0.952977 1.888574 1.888574 1.000 0.957136 1.000000 0.963467 0.018886 0.037771 88.857413 88.857413 0.037771
2 3 0.03 0.944715 1.888574 1.888574 1.000 0.949159 1.000000 0.958697 0.018886 0.056657 88.857413 88.857413 0.056657
3 4 0.04 0.937715 1.888574 1.888574 1.000 0.940997 1.000000 0.954272 0.018886 0.075543 88.857413 88.857413 0.075543
4 5 0.05 0.932483 1.888574 1.888574 1.000 0.935121 1.000000 0.950442 0.018886 0.094429 88.857413 88.857413 0.094429
5 6 0.10 0.900369 1.888574 1.888574 1.000 0.916547 1.000000 0.933494 0.094429 0.188857 88.857413 88.857413 0.188857
6 7 0.15 0.871008 1.888574 1.888574 1.000 0.885362 1.000000 0.917450 0.094429 0.283286 88.857413 88.857413 0.283286
7 8 0.20 0.840440 1.884797 1.887630 0.998 0.855393 0.999500 0.901936 0.094240 0.377526 88.479698 88.762984 0.377313
8 9 0.30 0.768186 1.860246 1.878502 0.985 0.805377 0.994667 0.869750 0.186025 0.563551 86.024551 87.850173 0.560150
9 10 0.40 0.677540 1.794145 1.857413 0.950 0.725652 0.983500 0.833725 0.179415 0.742965 79.414542 85.741265 0.728937
10 11 0.50 0.557894 1.537299 1.793390 0.814 0.621316 0.949600 0.791243 0.153730 0.896695 53.729934 79.338999 0.843135
11 12 0.60 0.413121 0.838527 1.634246 0.444 0.488311 0.865333 0.740755 0.083853 0.980548 -16.147309 63.424614 0.808815
12 13 0.70 0.292836 0.168083 1.424794 0.089 0.349117 0.754429 0.684806 0.016808 0.997356 -83.191690 42.479428 0.632000
13 14 0.80 0.207750 0.026440 1.250000 0.014 0.248701 0.661875 0.630293 0.002644 1.000000 -97.355996 25.000000 0.425080
14 15 0.90 0.133318 0.000000 1.111111 0.000 0.172087 0.588333 0.579381 0.000000 1.000000 -100.000000 11.111111 0.212540
15 16 1.00 0.001631 0.000000 1.000000 0.000 0.082915 0.529500 0.529735 0.000000 1.000000 -100.000000 0.000000 0.000000

ModelMetricsBinomialGLM: stackedensemble
** Reported on cross-validation data. **

MSE: 0.18681078669480172
RMSE: 0.4322161342370293
LogLoss: 0.5506562660945903
Null degrees of freedom: 9999
Residual degrees of freedom: 9989
Null deviance: 13828.721960196306
Residual deviance: 11013.12532189181
AIC: 11035.12532189181
AUC: 0.7886368448569471
AUCPR: 0.8074041154411439
Gini: 0.5772736897138941

Confusion Matrix (Act/Pred) for max f1 @ threshold = 0.3302404423659255: 
0 1 Error Rate
0 0 2014.0 2691.0 0.5719 (2691.0/4705.0)
1 1 513.0 4782.0 0.0969 (513.0/5295.0)
2 Total 2527.0 7473.0 0.3204 (3204.0/10000.0)
Maximum Metrics: Maximum metrics at their respective thresholds
metric threshold value idx
0 max f1 0.330240 0.749060 280.0
1 max f2 0.165847 0.860468 348.0
2 max f0point5 0.642005 0.739591 146.0
3 max accuracy 0.546801 0.713400 186.0
4 max precision 0.984111 1.000000 0.0
5 max recall 0.056564 1.000000 389.0
6 max specificity 0.984111 1.000000 0.0
7 max absolute_mcc 0.546801 0.429073 186.0
8 max min_per_class_accuracy 0.523329 0.711583 195.0
9 max mean_per_class_accuracy 0.546801 0.714850 186.0
10 max tns 0.984111 4705.000000 0.0
11 max fns 0.984111 5293.000000 0.0
12 max fps 0.007313 4705.000000 399.0
13 max tps 0.056564 5295.000000 389.0
14 max tnr 0.984111 1.000000 0.0
15 max fnr 0.984111 0.999622 0.0
16 max fpr 0.007313 1.000000 399.0
17 max tpr 0.056564 1.000000 389.0
Gains/Lift Table: Avg response rate: 52.95 %, avg score: 52.95 %
group cumulative_data_fraction lower_threshold lift cumulative_lift response_rate score cumulative_response_rate cumulative_score capture_rate cumulative_capture_rate gain cumulative_gain kolmogorov_smirnov
0 1 0.01 0.945921 1.869688 1.869688 0.990 0.959120 0.990000 0.959120 0.018697 0.018697 86.968839 86.968839 0.018484
1 2 0.02 0.934097 1.831917 1.850803 0.970 0.939804 0.980000 0.949462 0.018319 0.037016 83.191690 85.080264 0.036166
2 3 0.03 0.925157 1.813031 1.838212 0.960 0.929742 0.973333 0.942889 0.018130 0.055146 81.303116 83.821215 0.053446
3 4 0.04 0.915566 1.794145 1.827195 0.950 0.920021 0.967500 0.937172 0.017941 0.073088 79.414542 82.719547 0.070325
4 5 0.05 0.906642 1.813031 1.824363 0.960 0.911190 0.966000 0.931975 0.018130 0.091218 81.303116 82.436261 0.087605
5 6 0.10 0.866137 1.654391 1.739377 0.876 0.886109 0.921000 0.909042 0.082720 0.173938 65.439093 73.937677 0.157147
6 7 0.15 0.823701 1.631728 1.703494 0.864 0.845019 0.902000 0.887701 0.081586 0.255524 63.172805 70.349386 0.224281
7 8 0.20 0.785170 1.529745 1.660057 0.810 0.804308 0.879000 0.866853 0.076487 0.332011 52.974504 66.005666 0.280577
8 9 0.30 0.702005 1.395656 1.571923 0.739 0.743187 0.832333 0.825631 0.139566 0.471577 39.565628 57.192320 0.364669
9 10 0.40 0.618475 1.212465 1.482059 0.642 0.661145 0.784750 0.784509 0.121246 0.592823 21.246459 48.205855 0.409827
10 11 0.50 0.534506 1.078376 1.401322 0.571 0.576245 0.742000 0.742857 0.107838 0.700661 7.837583 40.132200 0.426485
11 12 0.60 0.448233 0.881964 1.314762 0.467 0.490541 0.696167 0.700804 0.088196 0.788857 -11.803588 31.476235 0.401397
12 13 0.70 0.366108 0.798867 1.241063 0.423 0.408348 0.657143 0.659024 0.079887 0.868744 -20.113314 24.106300 0.358648
13 14 0.80 0.283711 0.619452 1.163362 0.328 0.325523 0.616000 0.617337 0.061945 0.930689 -38.054769 16.336166 0.277767
14 15 0.90 0.186858 0.498584 1.089497 0.264 0.239080 0.576889 0.575308 0.049858 0.980548 -50.141643 8.949743 0.171196
15 16 1.00 0.007313 0.194523 1.000000 0.103 0.117545 0.529500 0.529532 0.019452 1.000000 -80.547686 0.000000 0.000000

Out[ ]:

We'll use the Combined Cycle Power Plant dataset. The goal here is to predict the energy output (in megawatts), given the temperature, ambient pressure, relative humidity and exhaust vacuum values. In this demo, you will use H2O's AutoML to outperform the state of the art results on this task.

In [ ]:
import os 
data_path = "https://github.com/h2oai/h2o-tutorials/raw/master/h2o-world-2017/automl/data/powerplant_output.csv"

# Load data into H2O
df = h2o.import_file(data_path)
Parse progress: |████████████████████████████████████████████████████████████████| (done) 100%

Exersice 3

A) Use .describe() to get a sense of what the data looks like

B) Split your dataframe in 80% train and 20% test

C) Run H2OAutoML of your dataframe and set 'HourlyEnergyOutputMW' as y. Tip: Set max_runtime_secs to 120 to avoid training for a really long time.

D) Check the leaderboard to see what model performed best.

In [ ]:
df.describe()
Rows:9568
Cols:5


TemperatureCelcius ExhaustVacuumHg AmbientPressureMillibar RelativeHumidity HourlyEnergyOutputMW
type real real real real real
mins 1.81 25.36 992.89 25.56 420.26
mean 19.651231187290957 54.3058037207358 1013.2590781772578 73.30897784280936 454.36500940635455
maxs 37.11 81.56 1033.3 100.16 495.76
sigma 7.452473229611082 12.7078929983268075.93878370581162 14.60026875672895717.066994999803423
zeros 0 0 0 0 0
missing0 0 0 0 0
0 14.96 41.76 1024.07 73.17 463.26
1 25.18 62.96 1020.04 59.08 444.37
2 5.11 39.4 1012.16 92.14 488.56
3 20.86 57.32 1010.24 76.64 446.48
4 10.82 37.5 1009.23 96.62 473.9
5 26.27 59.44 1012.23 58.77 443.67
6 15.89 43.96 1014.02 75.24 467.35
7 9.48 44.71 1019.12 66.43 478.42
8 14.64 45.0 1021.78 41.25 475.98
9 11.74 43.56 1015.14 70.72 477.5
In [ ]:
splits = df.split_frame(ratios = [0.8], seed = 1)
train = splits[0]
test = splits[1]
In [ ]:
aml = H2OAutoML(max_runtime_secs = 60, seed = 1, project_name = "powerplant_lb_frame")
aml.train(y = 'HourlyEnergyOutputMW', training_frame = train, leaderboard_frame = test)
AutoML progress: |███████████████████████████████████████████████████████████████| (done) 100%
Model Details
=============
H2OXGBoostEstimator :  XGBoost
Model Key:  XGBoost_2_AutoML_2_20211027_91229


Model Summary: 
number_of_trees
0 50.0

ModelMetricsRegression: xgboost
** Reported on train data. **

MSE: 3.747836610926001
RMSE: 1.9359330078610677
MAE: 1.423730124498044
RMSLE: 0.00427536303688152
Mean Residual Deviance: 3.747836610926001

ModelMetricsRegression: xgboost
** Reported on cross-validation data. **

MSE: 11.821639882202891
RMSE: 3.438261171319435
MAE: 2.4955980589850366
RMSLE: 0.007548090012687083
Mean Residual Deviance: 11.821639882202891

Cross-Validation Metrics Summary: 
mean sd cv_1_valid cv_2_valid cv_3_valid cv_4_valid cv_5_valid
0 mae 2.495596 0.027664 2.515636 2.493175 2.493330 2.523490 2.452347
1 mean_residual_deviance 11.821642 0.954860 12.802945 10.824024 11.812403 12.753228 10.915610
2 mse 11.821642 0.954860 12.802945 10.824024 11.812403 12.753228 10.915610
3 r2 0.959340 0.003248 0.956005 0.963346 0.959117 0.956391 0.961844
4 residual_deviance 11.821642 0.954860 12.802945 10.824024 11.812403 12.753228 10.915610
5 rmse 3.436014 0.138964 3.578120 3.289989 3.436917 3.571166 3.303878
6 rmsle 0.007544 0.000293 0.007838 0.007238 0.007511 0.007849 0.007281
Scoring History: 
timestamp duration number_of_trees training_rmse training_mae training_deviance
0 2021-10-27 09:13:04 3.936 sec 0.0 454.182071 453.861607 206281.353311
1 2021-10-27 09:13:04 3.957 sec 5.0 76.613804 76.325129 5869.674977
2 2021-10-27 09:13:04 3.976 sec 10.0 13.622706 12.907165 185.578105
3 2021-10-27 09:13:04 4.006 sec 15.0 4.123701 3.231729 17.004911
4 2021-10-27 09:13:04 4.084 sec 20.0 3.051780 2.291300 9.313360
5 2021-10-27 09:13:04 4.282 sec 25.0 2.726507 2.033951 7.433838
6 2021-10-27 09:13:04 4.332 sec 30.0 2.447299 1.808319 5.989271
7 2021-10-27 09:13:04 4.378 sec 35.0 2.270690 1.674315 5.156034
8 2021-10-27 09:13:04 4.431 sec 40.0 2.154064 1.587145 4.639993
9 2021-10-27 09:13:04 4.486 sec 45.0 2.034280 1.497311 4.138295
10 2021-10-27 09:13:04 4.546 sec 50.0 1.935933 1.423730 3.747837
Variable Importances: 
variable relative_importance scaled_importance percentage
0 TemperatureCelcius 1.744027e+06 1.000000 0.822740
1 ExhaustVacuumHg 2.920002e+05 0.167429 0.137750
2 AmbientPressureMillibar 4.533601e+04 0.025995 0.021387
3 RelativeHumidity 3.841553e+04 0.022027 0.018122
Out[ ]:

In [ ]:
aml.leaderboard.head()
model_id mean_residual_deviance rmse mse mae rmsle
XGBoost_2_AutoML_2_20211027_91229 10.85373.2945 10.85372.363910.00726642
XGBoost_1_AutoML_2_20211027_91229 11.59453.4050711.59452.416180.0075004
GBM_4_AutoML_2_20211027_91229 11.71213.4222911.71212.449940.00753313
GBM_3_AutoML_2_20211027_91229 11.94943.4567911.94942.4589 0.00760845
GBM_2_AutoML_2_20211027_91229 12.736 3.5687612.736 2.591960.00784773
GBM_1_AutoML_2_20211027_91229 13.11123.6209413.11122.5901 0.00796505
DRF_1_AutoML_2_20211027_91229 13.40913.6618513.40912.621280.00806712
XGBoost_3_AutoML_2_20211027_91229 16.524 4.0649716.524 3.037710.00892471
GLM_1_AutoML_2_20211027_91229 21.05354.5884121.05353.604910.0100956
XRT_1_AutoML_2_20211027_91229 32.13925.6691532.13923.876080.0124139
Out[ ]:

Exersice 4

A) Use the model you trained in exersice 3 to predict the HourlyEnergyOutputMW in your test set.

B) H2O has a function called model_perfomance to evaluate your model on test data. Use it to see how well your model predicts the test data.

In [ ]:
pred = aml.predict(test)
xgboost prediction progress: |███████████████████████████████████████████████████| (done) 100%
In [ ]:
perf = aml.leader.model_performance(test)
perf
ModelMetricsRegression: xgboost
** Reported on test data. **

MSE: 10.853732223779117
RMSE: 3.2945002995566894
MAE: 2.3639081756252893
RMSLE: 0.007266424462867062
Mean Residual Deviance: 10.853732223779117
Out[ ]:

The last AutoML library we want to show you is GAMA (General Automated Machine learning Assistent)

In [ ]:
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.metrics import log_loss, accuracy_score
from gama import GamaClassifier

if __name__ == "__main__":
    X, y = load_breast_cancer(return_X_y=True)
    X_train, X_test, y_train, y_test = train_test_split(
        X, y, stratify=y, random_state=0
    )

    automl = GamaClassifier(max_total_time=180, store="nothing", n_jobs=1)
    print("Starting `fit` which will take roughly 3 minutes.")
    automl.fit(X_train, y_train)

    label_predictions = automl.predict(X_test)
    probability_predictions = automl.predict_proba(X_test)

    print("accuracy:", accuracy_score(y_test, label_predictions))
    print("log loss:", log_loss(y_test, probability_predictions))
/usr/local/lib/python3.7/dist-packages/statsmodels/tools/_testing.py:19: FutureWarning: pandas.util.testing is deprecated. Use the functions in the public API at pandas.testing instead.
  import pandas.util.testing as tm
Starting `fit` which will take roughly 3 minutes.
accuracy: 0.9440559440559441
log loss: 0.13028902925742408

Exersice 5

A) Take the Combined Cycle Power Plant dataset from the previous assignemnt and the Digits dataset for the first exercise and use GAMA with the same prediction goals.

B) Evaluate the difference in the resulting metrics, but more over in pipeline structure. Which is your favorite AutoML library?

In [ ]:
 

Part 2 Transfer Learning

First we will have a look at the huggingface library. This is a library full of large pretrained models that are easily to be installed and used. Huggingface has a large amount of NLP (Natural Language Processing) algorithms but also offers alorithms for audio and vision processing. Check out their site for all the available models: https://huggingface.co/models

Here below we give an example of an algorithm named "bert-base-NER". bert-base-NER is a fine-tuned BERT model that is ready to use for Named Entity Recognition and achieves state-of-the-art performance for the NER task (Named Entity Recognition). It has been trained to recognize four types of entities: location (LOC), organizations (ORG), person (PER) and Miscellaneous (MISC).

For more info on named entity recognition you can check out this paper https://aclanthology.org/W03-0419.pdf

In [ ]:
from transformers import AutoTokenizer, AutoModelForTokenClassification
from transformers import pipeline

tokenizer = AutoTokenizer.from_pretrained("dslim/bert-base-NER")
model = AutoModelForTokenClassification.from_pretrained("dslim/bert-base-NER")

nlp = pipeline("ner", model=model, tokenizer=tokenizer)
example = "I'm Dorian from Utrecht and I work for Fruitpunch AI."

ner_results = nlp(example)
print(ner_results)
[{'entity': 'B-PER', 'score': 0.9955351, 'index': 4, 'word': 'Dorian', 'start': 4, 'end': 10}, {'entity': 'B-LOC', 'score': 0.9947187, 'index': 6, 'word': 'Utrecht', 'start': 16, 'end': 23}, {'entity': 'B-ORG', 'score': 0.9983479, 'index': 11, 'word': 'Fruit', 'start': 39, 'end': 44}, {'entity': 'I-ORG', 'score': 0.9977129, 'index': 12, 'word': '##pu', 'start': 44, 'end': 46}, {'entity': 'I-ORG', 'score': 0.98680216, 'index': 13, 'word': '##nch', 'start': 46, 'end': 49}, {'entity': 'I-ORG', 'score': 0.9877681, 'index': 14, 'word': 'AI', 'start': 50, 'end': 52}]

Exercise 1

A) Load the reviews dataset from one of our previous ai-code sessions.

B) The code below displays how to build an text classfier from scratch. Run the code and check the highest reached accuracy. (This can take ~5 minutes)

C) Search the huggingface library for a model that can classify the reviews in this dataset on positive or negative. And run it on the data.

D) Eveluate your transfered model. Does it outperform the models build from scratch?

In [ ]:
!git clone https://github.com/fruitpunch-ai-code/epoch-14.git
Cloning into 'epoch-14'...
remote: Enumerating objects: 35, done.
remote: Counting objects: 100% (35/35), done.
remote: Compressing objects: 100% (28/28), done.
remote: Total 35 (delta 5), reused 0 (delta 0), pack-reused 0
Unpacking objects: 100% (35/35), done.
In [ ]:
#required libraries
import pandas as pd
import numpy as np
from nltk.tokenize import word_tokenize
from nltk import pos_tag
from nltk.corpus import stopwords
from nltk.stem import WordNetLemmatizer
from collections import defaultdict
from nltk.corpus import wordnet as wn
from sklearn import model_selection
In [ ]:
import nltk
nltk.download('averaged_perceptron_tagger')
nltk.download('stopwords')
nltk.download('punkt')
nltk.download('wordnet')
#Set Random seed
np.random.seed(500)
[nltk_data] Downloading package averaged_perceptron_tagger to
[nltk_data]     /root/nltk_data...
[nltk_data]   Unzipping taggers/averaged_perceptron_tagger.zip.
[nltk_data] Downloading package stopwords to /root/nltk_data...
[nltk_data]   Unzipping corpora/stopwords.zip.
[nltk_data] Downloading package punkt to /root/nltk_data...
[nltk_data]   Unzipping tokenizers/punkt.zip.
[nltk_data] Downloading package wordnet to /root/nltk_data...
[nltk_data]   Unzipping corpora/wordnet.zip.
In [ ]:
df = pd.read_csv('/content/epoch-14/Challenges/reviews.csv', encoding='latin-1')
df.head()
Out[ ]:
text label
0 Stuning even for the non-gamer: This sound tr... __label__2
1 The best soundtrack ever to anything.: I'm re... __label__2
2 Amazing!: This soundtrack is my favorite musi... __label__2
3 Excellent Soundtrack: I truly like this sound... __label__2
4 Remember, Pull Your Jaw Off The Floor After H... __label__2
In [ ]:
# Step 1: Data Pre-processing - This will help in getting better results through the classification algorithms
Corpus = df.copy()
# Step 1a : Remove blank rows if any.
Corpus['text'].dropna(inplace=True)

# Step - 1b : Change all the text to lower case. This is required as python interprets 'dog' and 'DOG' differently
Corpus['text'] = [entry.lower() for entry in Corpus['text']]

# Step - 1c : Tokenization : In this each entry in the corpus will be broken into set of words
Corpus['text']= [word_tokenize(entry) for entry in Corpus['text']]

# Step - 1d : Remove Stop words, Non-Numeric and perfom Word Stemming/Lemmenting.

# WordNetLemmatizer requires Pos tags to understand if the word is noun or verb or adjective etc. By default it is set to Noun
tag_map = defaultdict(lambda : wn.NOUN)
tag_map['J'] = wn.ADJ
tag_map['V'] = wn.VERB
tag_map['R'] = wn.ADV

for index,entry in enumerate(Corpus['text']):
    # Declaring Empty List to store the words that follow the rules for this step
    Final_words = []
    # Initializing WordNetLemmatizer()
    word_Lemmatized = WordNetLemmatizer()
    # pos_tag function below will provide the 'tag' i.e if the word is Noun(N) or Verb(V) or something else.
    for word, tag in pos_tag(entry):
        # Below condition is to check for Stop words and consider only alphabets
        if word not in stopwords.words('english') and word.isalpha():
            word_Final = word_Lemmatized.lemmatize(word,tag_map[tag[0]])
            Final_words.append(word_Final)
    # The final processed set of words for each iteration will be stored in 'text_final'
    Corpus.loc[index,'text_final'] = str(Final_words)
In [ ]:
# Step 2: Split the model into Train and Test Data set
Train_X, Test_X, Train_Y, Test_Y = model_selection.train_test_split(Corpus['text_final'],Corpus['label'],test_size=0.3)
In [ ]:
# Step 3: Label encode the target variable  - This is done to transform Categorical data of string type in the data set into numerical values
from sklearn.preprocessing import LabelEncoder
Encoder = LabelEncoder()
Train_Y = Encoder.fit_transform(Train_Y)
Test_Y = Encoder.fit_transform(Test_Y)
In [ ]:
# Step 4: Vectorize the words by using TF-IDF Vectorizer - This is done to find how important a word in document is in comparison to the corpus
from sklearn.feature_extraction.text import TfidfVectorizer
Tfidf_vect = TfidfVectorizer(max_features=5000)
Tfidf_vect.fit(Corpus['text_final'])
Train_X_Tfidf = Tfidf_vect.transform(Train_X)
Test_X_Tfidf = Tfidf_vect.transform(Test_X)
In [ ]:
# Step 5: Now run ML algorithm to classify the text

# Classifier - Algorithm - Naive Bayes
from sklearn import naive_bayes
from sklearn.metrics import accuracy_score
# fit the training dataset on the classifier
Naive = naive_bayes.MultinomialNB()
Naive.fit(Train_X_Tfidf,Train_Y)
# predict the labels on validation dataset
predictions_NB = Naive.predict(Test_X_Tfidf)
# Use accuracy_score function to get the accuracy
print("Naive Bayes Accuracy Score -> ",accuracy_score(predictions_NB, Test_Y)*100)

# Classifier - Algorithm - SVM
from sklearn import svm
# fit the training dataset on the classifier
SVM = svm.SVC(C=1.0, kernel='linear', degree=3, gamma='auto')
SVM.fit(Train_X_Tfidf,Train_Y)
# predict the labels on validation dataset
predictions_SVM = SVM.predict(Test_X_Tfidf)
# Use accuracy_score function to get the accuracy
print("SVM Accuracy Score -> ",accuracy_score(predictions_SVM, Test_Y)*100)

# Classifier - Algorithm - Random Forest Classifier
from sklearn.ensemble import RandomForestClassifier
# fit the training dataset on the classifier
clf = RandomForestClassifier(n_estimators=400, max_depth=20, random_state=0)
clf.fit(Train_X_Tfidf,Train_Y)
# predict the labels on validation dataset
predictions_clf = clf.predict(Test_X_Tfidf)
# Use accuracy_score function to get the accuracy
print("Random Forest Classifier Accuracy Score -> ",accuracy_score(predictions_clf, Test_Y)*100)

# Classifier - Algorithm - AdaBoost Classifier
from sklearn.ensemble import AdaBoostClassifier
# fit the training dataset on the classifier
adaclf = AdaBoostClassifier(n_estimators=800, random_state=0)
adaclf.fit(Train_X_Tfidf,Train_Y)
# predict the labels on validation dataset
predictions_adaclf = adaclf.predict(Test_X_Tfidf)
# Use accuracy_score function to get the accuracy
print("AdaBoost Classifier Accuracy Score -> ",accuracy_score(predictions_adaclf, Test_Y)*100)

# Classifier - Algorithm - Linear SVC
from sklearn.svm import LinearSVC
# fit the training dataset on the classifier
svc = LinearSVC(random_state=0, tol=1e-5)
svc.fit(Train_X_Tfidf,Train_Y)
# predict the labels on validation dataset
predictions_svc = svc.predict(Test_X_Tfidf)
# Use accuracy_score function to get the accuracy
print("Linear SVC Accuracy Score -> ",accuracy_score(predictions_svc, Test_Y)*100)

# Classifier - Algorithm - Logistic Regression
from sklearn.linear_model import LogisticRegression
# fit the training dataset on the classifier
lr = LogisticRegression()
lr.fit(Train_X_Tfidf,Train_Y)
# predict the labels on validation dataset
predictions_lr = lr.predict(Test_X_Tfidf)
# Use accuracy_score function to get the accuracy
print("Logistic Regression Accuracy Score -> ",accuracy_score(predictions_lr, Test_Y)*100)

# Classifier - Algorithm - MLP Classifier
from sklearn.neural_network import MLPClassifier
# fit the training dataset on the classifier
mlp = MLPClassifier(hidden_layer_sizes=(13,13,13),max_iter=500)
mlp.fit(Train_X_Tfidf,Train_Y)
# predict the labels on validation dataset
predictions_nn = mlp.predict(Test_X_Tfidf)
# Use accuracy_score function to get the accuracy
print("MLPClassifier Accuracy Score -> ",accuracy_score(predictions_nn, Test_Y)*100)
Naive Bayes Accuracy Score ->  83.1
SVM Accuracy Score ->  84.7
Random Forest Classifier Accuracy Score ->  81.89999999999999
AdaBoost Classifier Accuracy Score ->  80.03333333333333
Linear SVC Accuracy Score ->  84.03333333333333
Logistic Regression Accuracy Score ->  85.2
MLPClassifier Accuracy Score ->  81.83333333333334
In [ ]:
from transformers import AutoModelForSequenceClassification, AutoTokenizer, AutoModelForQuestionAnswering, pipeline

model_name = "aychang/roberta-base-imdb"

model = AutoModelForQuestionAnswering.from_pretrained(model_name)
tokenizer = AutoTokenizer.from_pretrained(model_name)

nlp = pipeline("sentiment-analysis", model=model_name, tokenizer=model_name)
Some weights of the model checkpoint at aychang/roberta-base-imdb were not used when initializing RobertaForQuestionAnswering: ['classifier.dense.weight', 'classifier.dense.bias', 'classifier.out_proj.bias', 'classifier.out_proj.weight']
- This IS expected if you are initializing RobertaForQuestionAnswering from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing RobertaForQuestionAnswering from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Some weights of RobertaForQuestionAnswering were not initialized from the model checkpoint at aychang/roberta-base-imdb and are newly initialized: ['qa_outputs.weight', 'qa_outputs.bias']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
In [ ]:
# nlp only takes lists as input
list_of_text = df.text.tolist()
In [ ]:
# To speed things up a little bit grab a subset of the data
results = nlp(list_of_text[0:1000])
/usr/local/lib/python3.7/dist-packages/torch/utils/data/dataloader.py:481: UserWarning: This DataLoader will create 8 worker processes in total. Our suggested max number of worker in current system is 2, which is smaller than what this DataLoader is going to create. Please be aware that excessive worker creation might get DataLoader running slow or even freeze, lower the worker number to avoid potential slowness/freeze if necessary.
  cpuset_checked))
In [ ]:
# extract the predictions from the results
pred = []
for i in range(1000):
  if results[i]['label'] == 'pos':
    pred.append(1)
  if results[i]['label'] == 'neg':
    pred.append(0)
In [ ]:
# Calculate accuracy score but first transform labels to numbers
df['label'] = Encoder.fit_transform(df['label'])
print("Transfered model Accuracy Score -> ",accuracy_score(pred, df['label'][0:1000])*100)
Transfered model Accuracy Score ->  92.80000000000001

Exersize 2: Transfer learning for object detection

Besides libaries like huggingface there are many other models you can find a trained version from online. These can be huge networks that have been trained for months on a full datacenter with the newest GPUs like GTP 2 and 3, or smaller architectrues like ResNet50 or VGG16. While these models are doable to train from scratch yourself, you might not have enough data or simply would like to reduce training time, cost or evironmental impact. These models come with a set op pretrained weights and you can then transfer learn your own dataset using the experience of the pretrained model.

In this exercize we will be looking at the YOLO (You Only Look Once) object detection model. While there are more advanced object decteion models like Facebook's Detectron2 (RCNN based) and DETR (Transformer based) models, these are huge models which still take a long time to transfer learn. Yolo's fifth version provides state of the art performance in relatively small models (in different size verarieties) which makes it very useful for edge detection cases and cases where the inference speed or used compute resources are relevant.

Firts we will be downloading the COCO2017 dataset, which is one of the most commono object detection benschmarking datasets. Then we will download Yolov5 and get to the exersize.

In [ ]:
# Download COCO test-dev2017, will take a couple of minutes
!wget https://ultralytics.com/assets/coco2017labels.zip 
!unzip -q coco2017labels.zip -d ../datasets && rm coco2017labels.zip
!f="test2017.zip" && curl http://images.cocodataset.org/zips/$f -o $f && unzip -q $f -d ../datasets/coco/images
--2021-11-04 11:58:07--  https://ultralytics.com/assets/coco2017labels.zip
Resolving ultralytics.com (ultralytics.com)... 151.101.1.195, 151.101.65.195
Connecting to ultralytics.com (ultralytics.com)|151.101.1.195|:443... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: https://github.com/ultralytics/yolov5/releases/download/v1.0/coco2017labels.zip [following]
--2021-11-04 11:58:07--  https://github.com/ultralytics/yolov5/releases/download/v1.0/coco2017labels.zip
Resolving github.com (github.com)... 140.82.112.4
Connecting to github.com (github.com)|140.82.112.4|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://github-releases.githubusercontent.com/264818686/f7319580-68c5-11eb-95a4-9eb6453913da?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20211104%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20211104T115808Z&X-Amz-Expires=300&X-Amz-Signature=9a49365435314262b079f9670def3519b356212565e1896cdecc408019775075&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=264818686&response-content-disposition=attachment%3B%20filename%3Dcoco2017labels.zip&response-content-type=application%2Foctet-stream [following]
--2021-11-04 11:58:08--  https://github-releases.githubusercontent.com/264818686/f7319580-68c5-11eb-95a4-9eb6453913da?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20211104%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20211104T115808Z&X-Amz-Expires=300&X-Amz-Signature=9a49365435314262b079f9670def3519b356212565e1896cdecc408019775075&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=264818686&response-content-disposition=attachment%3B%20filename%3Dcoco2017labels.zip&response-content-type=application%2Foctet-stream
Resolving github-releases.githubusercontent.com (github-releases.githubusercontent.com)... 185.199.108.154, 185.199.109.154, 185.199.110.154, ...
Connecting to github-releases.githubusercontent.com (github-releases.githubusercontent.com)|185.199.108.154|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 71005511 (68M) [application/octet-stream]
Saving to: ‘coco2017labels.zip’

coco2017labels.zip  100%[===================>]  67.72M  55.0MB/s    in 1.2s    

2021-11-04 11:58:09 (55.0 MB/s) - ‘coco2017labels.zip’ saved [71005511/71005511]

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 6339M  100 6339M    0     0  49.8M      0  0:02:07  0:02:07 --:--:-- 30.9M
In [ ]:
# Download yolo5 and install its dependencies
!git clone https://github.com/ultralytics/yolov5  # clone repo
!pip install -qr yolov5/requirements.txt  # install dependencies
Cloning into 'yolov5'...
remote: Enumerating objects: 9899, done.
remote: Counting objects: 100% (86/86), done.
remote: Compressing objects: 100% (64/64), done.
remote: Total 9899 (delta 41), reused 45 (delta 22), pack-reused 9813
Receiving objects: 100% (9899/9899), 10.31 MiB | 25.07 MiB/s, done.
Resolving deltas: 100% (6833/6833), done.
     |████████████████████████████████| 11.3 MB 5.3 MB/s 
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
google-colab 1.0.0 requires pandas~=1.1.0; python_version >= "3.0", but you have pandas 1.3.4 which is incompatible.
gama 21.0.0 requires pandas<1.1,>=1.0, but you have pandas 1.3.4 which is incompatible.
In [ ]:
 

A) Train a Yolov5 S variant on the data using the train.py script in the yolov5 folder with 3 epochs. (the data should already be in the right spot)

B) Go the the results folder in yolov5 folder and evaluate the results of the experiment you just ran

C) Select a few photos (from ../datasets/coco128/images/train2017) and use the detect.py script to detect the objects on it.

D) Choose a larger variant of the Yolov5 model and repeat training and detection on the same photos, are the results significantly better compared to the loger training time?

In [ ]:
!python yolov5/train.py --img 640 --batch 16 --epochs 3 --data coco128.yaml --weights yolov5s.pt --cache #--device 0
Downloading https://ultralytics.com/assets/Arial.ttf to /root/.config/Ultralytics/Arial.ttf...
train: weights=yolov5s.pt, cfg=, data=coco128.yaml, hyp=yolov5/data/hyps/hyp.scratch.yaml, epochs=3, batch_size=16, imgsz=640, rect=False, resume=False, nosave=False, noval=False, noautoanchor=False, evolve=None, bucket=, cache=ram, image_weights=False, device=, multi_scale=False, single_cls=False, adam=False, sync_bn=False, workers=8, project=yolov5/runs/train, name=exp, exist_ok=False, quad=False, linear_lr=False, label_smoothing=0.0, patience=100, freeze=0, save_period=-1, local_rank=-1, entity=None, upload_dataset=False, bbox_interval=-1, artifact_alias=latest
github: up to date with https://github.com/ultralytics/yolov5 ✅
YOLOv5 🚀 v6.0-54-gac2c49a torch 1.9.0+cu111 CUDA:0 (Tesla K80, 11441MiB)

hyperparameters: lr0=0.01, lrf=0.1, momentum=0.937, weight_decay=0.0005, warmup_epochs=3.0, warmup_momentum=0.8, warmup_bias_lr=0.1, box=0.05, cls=0.5, cls_pw=1.0, obj=1.0, obj_pw=1.0, iou_t=0.2, anchor_t=4.0, fl_gamma=0.0, hsv_h=0.015, hsv_s=0.7, hsv_v=0.4, degrees=0.0, translate=0.1, scale=0.5, shear=0.0, perspective=0.0, flipud=0.0, fliplr=0.5, mosaic=1.0, mixup=0.0, copy_paste=0.0
Weights & Biases: run 'pip install wandb' to automatically track and visualize YOLOv5 🚀 runs (RECOMMENDED)
TensorBoard: Start with 'tensorboard --logdir yolov5/runs/train', view at http://localhost:6006/

WARNING: Dataset not found, nonexistent paths: ['/datasets/coco128/images/train2017']
Downloading https://github.com/ultralytics/yolov5/releases/download/v1.0/coco128.zip to coco128.zip...
100% 6.66M/6.66M [00:00<00:00, 68.4MB/s]
Dataset autodownload success, saved to ../datasets

Downloading https://github.com/ultralytics/yolov5/releases/download/v6.0/yolov5s.pt to yolov5s.pt...
100% 14.0M/14.0M [00:00<00:00, 44.3MB/s]


                 from  n    params  module                                  arguments                     
  0                -1  1      3520  models.common.Conv                      [3, 32, 6, 2, 2]              
  1                -1  1     18560  models.common.Conv                      [32, 64, 3, 2]                
  2                -1  1     18816  models.common.C3                        [64, 64, 1]                   
  3                -1  1     73984  models.common.Conv                      [64, 128, 3, 2]               
  4                -1  2    115712  models.common.C3                        [128, 128, 2]                 
  5                -1  1    295424  models.common.Conv                      [128, 256, 3, 2]              
  6                -1  3    625152  models.common.C3                        [256, 256, 3]                 
  7                -1  1   1180672  models.common.Conv                      [256, 512, 3, 2]              
  8                -1  1   1182720  models.common.C3                        [512, 512, 1]                 
  9                -1  1    656896  models.common.SPPF                      [512, 512, 5]                 
 10                -1  1    131584  models.common.Conv                      [512, 256, 1, 1]              
 11                -1  1         0  torch.nn.modules.upsampling.Upsample    [None, 2, 'nearest']          
 12           [-1, 6]  1         0  models.common.Concat                    [1]                           
 13                -1  1    361984  models.common.C3                        [512, 256, 1, False]          
 14                -1  1     33024  models.common.Conv                      [256, 128, 1, 1]              
 15                -1  1         0  torch.nn.modules.upsampling.Upsample    [None, 2, 'nearest']          
 16           [-1, 4]  1         0  models.common.Concat                    [1]                           
 17                -1  1     90880  models.common.C3                        [256, 128, 1, False]          
 18                -1  1    147712  models.common.Conv                      [128, 128, 3, 2]              
 19          [-1, 14]  1         0  models.common.Concat                    [1]                           
 20                -1  1    296448  models.common.C3                        [256, 256, 1, False]          
 21                -1  1    590336  models.common.Conv                      [256, 256, 3, 2]              
 22          [-1, 10]  1         0  models.common.Concat                    [1]                           
 23                -1  1   1182720  models.common.C3                        [512, 512, 1, False]          
 24      [17, 20, 23]  1    229245  models.yolo.Detect                      [80, [[10, 13, 16, 30, 33, 23], [30, 61, 62, 45, 59, 119], [116, 90, 156, 198, 373, 326]], [128, 256, 512]]
Model Summary: 270 layers, 7235389 parameters, 7235389 gradients, 16.5 GFLOPs

Transferred 349/349 items from yolov5s.pt
Scaled weight_decay = 0.0005
optimizer: SGD with parameter groups 57 weight, 60 weight (no decay), 60 bias
albumentations: version 1.0.3 required by YOLOv5, but version 0.1.12 is currently installed
train: Scanning '../datasets/coco128/labels/train2017' images and labels...128 found, 0 missing, 2 empty, 0 corrupted: 100% 128/128 [00:00<00:00, 1087.99it/s]
train: New cache created: ../datasets/coco128/labels/train2017.cache
train: Caching images (0.1GB ram): 100% 128/128 [00:00<00:00, 252.08it/s]
val: Scanning '../datasets/coco128/labels/train2017.cache' images and labels... 128 found, 0 missing, 2 empty, 0 corrupted: 100% 128/128 [00:00<?, ?it/s]
val: Caching images (0.1GB ram): 100% 128/128 [00:01<00:00, 108.07it/s]
[W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
[W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
Plotting labels... 

autoanchor: Analyzing anchors... anchors/target = 4.27, Best Possible Recall (BPR) = 0.9935
Image sizes 640 train, 640 val
Using 2 dataloader workers
Logging results to yolov5/runs/train/exp
Starting training for 3 epochs...

     Epoch   gpu_mem       box       obj       cls    labels  img_size
       0/2     3.37G   0.04621   0.07109   0.02112       203       640: 100% 8/8 [00:12<00:00,  1.61s/it]
               Class     Images     Labels          P          R     mAP@.5 mAP@.5:.95: 100% 4/4 [00:02<00:00,  1.49it/s]
                 all        128        929      0.655      0.548      0.623      0.412

     Epoch   gpu_mem       box       obj       cls    labels  img_size
       1/2      4.5G   0.04564   0.06899   0.02116       143       640: 100% 8/8 [00:09<00:00,  1.18s/it]
               Class     Images     Labels          P          R     mAP@.5 mAP@.5:.95: 100% 4/4 [00:02<00:00,  1.48it/s]
                 all        128        929      0.679      0.554      0.633      0.419

     Epoch   gpu_mem       box       obj       cls    labels  img_size
       2/2      4.5G   0.04487   0.06882   0.01998       253       640: 100% 8/8 [00:09<00:00,  1.19s/it]
               Class     Images     Labels          P          R     mAP@.5 mAP@.5:.95: 100% 4/4 [00:02<00:00,  1.47it/s]
                 all        128        929      0.708      0.545      0.629      0.423

3 epochs completed in 0.012 hours.
Optimizer stripped from yolov5/runs/train/exp/weights/last.pt, 14.9MB
Optimizer stripped from yolov5/runs/train/exp/weights/best.pt, 14.9MB

Validating yolov5/runs/train/exp/weights/best.pt...
Fusing layers... 
Model Summary: 213 layers, 7225885 parameters, 0 gradients, 16.5 GFLOPs
               Class     Images     Labels          P          R     mAP@.5 mAP@.5:.95: 100% 4/4 [00:05<00:00,  1.50s/it]
                 all        128        929      0.709      0.545      0.629      0.423
              person        128        254      0.815      0.673      0.775      0.509
             bicycle        128          6      0.799      0.667      0.614      0.355
                 car        128         46      0.804      0.357      0.486      0.209
          motorcycle        128          5      0.702        0.6      0.791      0.583
            airplane        128          6          1      0.795      0.995      0.717
                 bus        128          7      0.655      0.714      0.719      0.605
               train        128          3      0.847          1      0.995      0.682
               truck        128         12      0.516       0.25      0.397      0.216
                boat        128          6      0.791      0.333      0.422      0.135
       traffic light        128         14      0.576      0.143       0.24      0.161
           stop sign        128          2      0.635        0.5      0.828      0.713
               bench        128          9       0.97      0.444      0.572      0.256
                bird        128         16      0.939      0.968      0.988      0.645
                 cat        128          4      0.984       0.75      0.822      0.694
                 dog        128          9      0.887      0.667      0.903       0.54
               horse        128          2      0.688          1      0.995      0.697
            elephant        128         17       0.96      0.882      0.943      0.681
                bear        128          1      0.546          1      0.995      0.995
               zebra        128          4       0.86          1      0.995      0.952
             giraffe        128          9      0.821      0.778      0.905       0.57
            backpack        128          6          1       0.31      0.457      0.195
            umbrella        128         18      0.775      0.576       0.74      0.418
             handbag        128         19      0.625      0.105      0.167       0.11
                 tie        128          7      0.957      0.571      0.701      0.441
            suitcase        128          4          1        0.9      0.995      0.621
             frisbee        128          5       0.64        0.8      0.798      0.664
                skis        128          1      0.626          1      0.995      0.497
           snowboard        128          7      0.986      0.714      0.768      0.556
         sports ball        128          6       0.67        0.5      0.579      0.342
                kite        128         10      0.632      0.517      0.598      0.221
        baseball bat        128          4      0.473      0.459      0.277      0.137
      baseball glove        128          7      0.456      0.429      0.334      0.182
          skateboard        128          5        0.7      0.481      0.736      0.548
       tennis racket        128          7      0.558      0.571      0.541      0.286
              bottle        128         18      0.605      0.389      0.485      0.282
          wine glass        128         16      0.719      0.812      0.786      0.382
                 cup        128         36      0.877      0.361      0.531      0.311
                fork        128          6      0.384      0.167      0.239      0.191
               knife        128         16      0.908      0.617      0.682      0.443
               spoon        128         22      0.835      0.364      0.526      0.261
                bowl        128         28      0.792      0.536      0.634      0.467
              banana        128          1          0          0      0.142     0.0995
            sandwich        128          2          0          0     0.0951     0.0717
              orange        128          4          1          0       0.67      0.317
            broccoli        128         11       0.34      0.182      0.283      0.243
              carrot        128         24      0.688       0.46      0.611      0.402
             hot dog        128          2      0.425      0.774      0.497      0.473
               pizza        128          5      0.622          1      0.824      0.551
               donut        128         14      0.702          1      0.952      0.853
                cake        128          4      0.733          1      0.945      0.777
               chair        128         35      0.519      0.493      0.489      0.223
               couch        128          6       0.68       0.36      0.746      0.406
        potted plant        128         14      0.796      0.714      0.808      0.482
                 bed        128          3          1          0      0.474      0.318
        dining table        128         13      0.852      0.445      0.478      0.315
              toilet        128          2      0.513        0.5      0.554      0.487
                  tv        128          2      0.754          1      0.995      0.945
              laptop        128          3          1          0       0.39      0.147
               mouse        128          2          1          0     0.0263      0.021
              remote        128          8      0.746      0.625      0.636      0.488
          cell phone        128          8      0.556      0.167      0.419      0.223
           microwave        128          3      0.404          1      0.995      0.732
                oven        128          5      0.369        0.4      0.432      0.249
                sink        128          6      0.356      0.167      0.269      0.147
        refrigerator        128          5      0.704        0.8      0.813       0.45
                book        128         29      0.626      0.138      0.298      0.135
               clock        128          9      0.856      0.778      0.893      0.574
                vase        128          2      0.241          1      0.663      0.622
            scissors        128          1          1          0     0.0207    0.00415
          teddy bear        128         21      0.844      0.381      0.623      0.345
          toothbrush        128          5      0.987        0.6      0.662       0.45
Results saved to yolov5/runs/train/exp
In [ ]:
!python yolov5/detect.py --source ../datasets/coco128/images/train2017/000000000009.jpg  --weights yolov5/yolov5s.pt --conf 0.4
detect: weights=['yolov5/yolov5s.pt'], source=../datasets/coco128/images/train2017/000000000009.jpg, imgsz=[640, 640], conf_thres=0.4, iou_thres=0.45, max_det=1000, device=, view_img=False, save_txt=False, save_conf=False, save_crop=False, nosave=False, classes=None, agnostic_nms=False, augment=False, visualize=False, update=False, project=yolov5/runs/detect, name=exp, exist_ok=False, line_thickness=3, hide_labels=False, hide_conf=False, half=False, dnn=False
YOLOv5 🚀 v6.0-54-gac2c49a torch 1.9.0+cu111 CUDA:0 (Tesla K80, 11441MiB)

Downloading https://github.com/ultralytics/yolov5/releases/download/v6.0/yolov5s.pt to yolov5/yolov5s.pt...
100% 14.0M/14.0M [00:00<00:00, 92.8MB/s]

Fusing layers... 
Model Summary: 213 layers, 7225885 parameters, 0 gradients
image 1/1 /datasets/coco128/images/train2017/000000000009.jpg: 480x640 1 bowl, 1 broccoli, Done. (0.025s)
Speed: 0.5ms pre-process, 25.4ms inference, 2.1ms NMS per image at shape (1, 3, 640, 640)
Results saved to yolov5/runs/detect/exp